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COST  UNCERTAINTY  ASSESSMENT  METHODOLOGY: 
A  CRITICAL  OVERVIEW 


K.  T.  Wallenius 
Clemson  Unlversitv 


OVERVIEW 


An  appraisal  of  the  quality  of  methodology 
and  software  for  performing  cost§  uncertainty 
analysis  will  be  given.  It  will  be  argued  that 

a  the  standard  practice  of  developing  a  "most 
likely  cost"  and  then  generating  an 
uncertainty  distribution  associated  with  that 
figure  is  a  consequence  of  historical 
precedent  and  places  the  cart  before  the 
horse.  Decision  theoretic  considerations 
require  that  uncertainty  assessment  precede 
total  cost  estimation.  This  fact  underscores 
the  prieacy  of  cost  uncertainty  analysis  as  a 
^  a  tool  in  the  overall  cost  analysis  process. 

methodology  for  encoding  subjective 
probability  distributions  are  out  of  date, 
being,  for  the  most  part,  modifications  of 
the  PERT  techniques  of  the  sixties.  These 
techniques,  ignoring  the  implications  of  the 
vast  behavioral  science  literature  on 
uncertainty  elicitation  and  calibration, 
sacrifice  human  cognitive  capability  on  the 
altar  of  mathematical  convenience.  This  may 
explain,  at  least  in  part,  the  general 
reluctance  of  cost  analysts  to  undertake  a 
serious  assessment  of  cost  uncertainty. 

§  The  term  "cost"  is  used  in  a  generic  way 
this  paper.  Its  meaning  will  depend  on  the  - 
purpose  of  the  analysis,  (e.g.  first  unit 
cost,  nonrecurring  cost,  LCC,  etc.)  j 
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9  methodology  for  processing  elemental 

uncertainty  assessments  into  uncertainty 
distributions  over  higher  order  structures 
(e.g.  subsystem  CER's  or  total  system  cost) 
often  ignores  important  source*  of 
variability.  This  may  explain  why 
uncar-ta4njtir'distr  ibutions  over  total  cost  are 
usually  too  tight.  CER  compendia  which  omit 
covariance  data  (see  reference  [4],  for 
example)  are  of  little  value  in  quantifying 
V  uncertainty . 

•  ^extant  computer  software  designed  to 
implement  cost  uncertainty  analysis 
methodology  is  often  poorly  written  and,  in 
some  cases,  may  lead  to  conclusions  which  are 

inconsistent  with  the  inputs.  While  "garbage  < 

in,  garbage  out"  is  axiomatic,  users  of 
computer  software  who  expand  the  time  and 
effort  to  generate  reliable  Inputs  have  the 

right  to  expect  output  fidelity.  —  „  „ 

THE  -TBIMCX-  QE  QQ3.T  qffCBmiETY  J&ALX21S  ■  y  '  :7V."  _  1 

Providing  an  answer  to  the  question  "How  much 
will  it  cost?"  is  Job  one  for  the  cost  analyst  ar.d 
has  a  long  historical  precedence  over  the  (more!) 
important  question  "How  sure  are  you  of  that 
figure?".  This  is  probably  due  to  the  facts  that 
it's  easier  to  come  up  with  a  single  number  than  a 
probability  distribution,  and,  budget*  of  the  past 
were  based  on  most  likely  costs,  Many  of  today's 

directives  require  a  quantification  of  cost  ! 

uncertainty  and  soma  budgeting  process**  allow 

component-program  cost  uncertainties  to  enter  into  > 

the  total  budget  calculations  (a.g.  TRACE).  Tha  ; 

form  of  thasa  directives  is  rsvsaling:  usually,  j 

thay  refar  to  tha  "uncartalnty  aasociatad  with  a  ! 

moat  likely  cost"  implying  tha  coat  aatimate  coeaa 

first.  The  modern  approach  [18]  to  decision  making 

dictates  that  uncertainty  considerations  co»e 

first.  It  is  one  of  tha  two  prerequisite  inputs 

nesded  for  dtvsloping  an  optimal  estimate  of  cost 

(ths  othsr  being  a  conaidaratlon  of  tha  relative 

seriousness  of  oversstlmates  and  underest lmates . ) 

The  problems  of  generating  an  uncertainty 


distribution  over  total  cost  are  discussed  below 
but,  for  now,  suppose  design  specifications, 
engineering  information,  contractor  performance 
requirements,  historical  data,  subjective 
assessments,  etc.  have  been  correctly  processed 
resulting  in  such  an  uncertainty  distribution.  As 
such,  this  distribution  summarizes  the  totality  of 
available  information.  Besides  answering 
important  probabilistic  questions,  this 
distribution  is  the  cornerstone  for  generating 
estimates  of  total  cost.  Standard  texts  on 
decision  making,  for  example  [18],  develop  optimal 
estimates  which  account  for  the  relative 
seriousness  of  the  difference  between  the  estimate 
and  the  actual  cost.  If  C  and  C  denote  the 
estimate  and  the  cost,  respectively,  and  e  ■  (?  -  C 
is  the  error  (  e>0  is  an  underestimate  or  a  cost 
overrun),  it  is  customary  to  code  the  error  ^ 
•eriouaneaa  in  terms  of  a  regret  function  R(C,C) 
having  the  properties  that  R  is  nonnegative,  is 
zero  when  a  =  0,  and  is  nondecreaelng  as  the  error 
moves  away  from  zero  in  either  direction.  Some 
typical  examples  are  shown  in  ficrura  1. 


? j-oURE  1  Some  Regret  Functions 

In  figure  la,  the  seriousness  of  an  error  is 
proportional  to  the  absolute  value  of  the  error 
with  underestimates  k  times  as  serious  as 
overestimates.  In  this  case,  the  optimal#  cost 


§  An  optimal  estimate  for  the  purposes  of  this 
paper  is  one  that  minimizes  expected  regret. 

There  are  other  definitions  of  optimal.  The  cost 
uncertainty  distribution  and  the  regret  function 
are  needed  for  most  reasonable  optimality  criteria 


estimate  Is  tha  k/(k+l)th  percentile  of  the 
uncertainty  distribution.  If  It  ■  3  for  example, 
the  75th  percentile  would  be  optimal.  In  figure 
lb,  the  seriousness  of  sn  error  is  proportional  to 
the  square  of  the  error.  In  this  case,  the 
optimal  estimate  is  the  mean  of  the  uncertainty 
distribution.  In  figure  lc,  the  regret  io  zero  if 
the  estimate  is  within  tolerance,  and  very  large 
otherwise.  A  regret  function  similar  to  this 
might  be  appropriate  when  errors  outside  a  given 
range  are  not  tolerable  (e.g.  h  ■  1.15  might  be 
used  to  protect  against  a  Nunn  Aamendment 
cancellation) .  In  this  case,  optimal  estimates 
depend  on  the  values  of  1,  L ,  n,  and  H  but  are 
near  the  mode  of  typical  unimodal  uncertainty 
distributions . 

Most  cost  uncertainty  directives  and 
technical  papers  use  the  term  "most  liicely  coat" 
referring,  presumably,  to  some  sort  of  modal 
value.  This  implicitly  says  something  universal 
about  the  relative  seriousness  of  estimation 
errors  which,  in  fact,  will  differ  between 
organizations  (perhaps  even  between  different 
levels  within  the  same  organization) .  The  point 
here  is  that  the  optimal  estimate  logically 
depends  on  its  intended  use.  The  cost  uncertainty 
distribution  is  of  fundamental  importance  in 
generating  the  optimal  estimate  and  is  the  job  of 
the  cost  analyst.  Generating  the  cost  estimate  is 
the  perogative  of  the  cognizant  command  and  should 
reflect  the  relative  seriousness  of  errors.  This 
functional  separation  allows  each  organizational 
unit  to  focus  on  tasks  best  suited  to  their 
expertise  and  responsibility. 

Thus,  the  cost  uncertainty  distribution 
should  not  be  an  afterthought,  tagging  along 
behind  the  point  estimate.  It  is  the  cornerstone 
upon  which  that  estimate  is  based.  So,  what  is 
the  state-of-the-art  in  coot  uncertainty 
assessment  methodology?  What  tools  (training 
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programs,  books,  computer  software,  documentation, 
etc.)  are  available  to  assist  cost  analysts  to 
perform  this  pivotal  ta^k?  It  is  not  feasible  to 
survey  the  entire  literature  in  a  paper  of  this 
type  nor  is  the  entire  literature  available  for 
such  a  survey  (viz.  proprietary  documents  and 
software).  Comment*  will  be  organized  around  two 
component  tasks:  (1)  quantifying  uncertainty  at 
the  most  minute  level  of  disaggregation,  (2) 
processing  of  these  "particle"  assessments  to 
produce  an  uncertainty  distribution  over  total 
cost , 


1 .  QUANTIFYING  UNCERTAINTY 

Disaggregation  of  complex  systems  into  cost 
components  allows  the  analyst  to  focus  on 
subproblems  with  relatively  few  variables  thus 
simplifying  individual  assessments.  (Of  course, 
this  requires  aggregation  of  these  component 
assessments  to  generate  a  total  cost  uncertainty 
distribution  but  that's  more  a  mathematical 
problem  than  a  cognitive  one.)  Suppose  the 
quantity  of  interest  is  dol lars-per-athe  of 
debugged  software  for  a  midcourse  guidance 
computer.  What  guidelines  and  decision  aid*  are 
available  to  help  the  cost  analyst  quantify 
uncertainty  about  this  component  coet?  If  he  has 
read  the  1984  state-of-the-art  survey  in  the 
Journal  of  Cost  Analysis  [15],  he  might  try  to  fit 
a  4-parameter  beta  distribution  by  specifying  3 
quantities:  a  mode  (m),  a  least  possible  coat  (1), 
and  a  highest  possible  coat  (h) .  A  fourth 
specif lcation  pertaining  to  the  spread  (a)  of  the 
distribution  is  also  required  to  mathematically 
nail  down  a  unique  member  of  the  beta  family.  The 
analyst  may  alleviate  the  cognitive  pain  of 
specifying  s  by  using  some  ad  hoc  nominal  value 
(e.g.  the  PERT  formula  for  s)  but,  by  so  doing,  a 
decision  about  a  is  nevertheless  made.  Three  of 
the  four  specifications,  (1,  h,and  s)  srs  quits 
difficult  for  ae  to  think  about  and  m  la  also  a 
little  unnatural.  It'e  hard  to  think  about  a 
number  h  such  that  there  is  positive  probability 
that  the  cost  can  get  arbitrarily  close  to  h  and, 
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•t  the  same  time,  feel  secure  in  betting  your  life 
against  •  Jellybean  that  the  cost  will  not  exceed 
h! 

Since  cost  element  uncertainties  must  be 
aggregated  to  generate  a  total  cost  uncertainty 
distribution,  it  is  necessary  to  express  them 
mathematically .  This  usually  takes  the  fora  of 
some  parametric  family  of  probability  density 
functions  such  as  the  beta,  gamma,  or  Weibull 
families.  Over  90*  of  the  cost  uncertainty 
literature  advocates  use  of  the  beta  family  for 
this  purpose.  See,  for  example  references  [1] 

[2],  [6],  [9],  [16],  [19],  and  [23].  The 

state-of-the-art  survey  paper  [15]  focuses 
exclusively  on  fitting  beta  distributions.  The 
beta  family  is  shape-rich  and  fairly  adaquate  in 
representing  unimodal  uncertainty  situations.  Its 
popularity  in  cost  analysis  is  due,  in  part,  to 
the  early  60 1 s  success  of  PERT  analysis  (basad  on 
beta  representations  of  completion  time 
uncertainty) .  Many  of  tha  112  raferancaa  citad  in 
the  bibliography  of  [15]  deal  with  PERT.  In  all 
the  beta  fitting  techniques  cited,  the  user  Is 
required  to  specify  1  and  h  plus  either  »  or  the 
mean  (mu) .  There  la  an  obvious  mathematical 
convenience  in  this  since  1  and  h  are  two  of  the 
four  parameters  needed.  m  (or  mu)  determines  a 
linear  relation  between  the  two  shape  parameter* 
so  that  Just  about  any  fourth  condition  makes 
solving  for  the  shape  parameters  relatively  easy. 
In  looking  at  the  papers  in  which  this  methodology 
was  developed,  mathematical  convenience  was  the 
dominate,  if  not  the  only,  consideration  in 
designing  the  subjective  input  requirements. 

But  people  have  to  do  it.  There  is  an 
enormous  literature  describing  how  people  make 
Judgments  undsr  uncertainty,  how  biases  and 
heuristics  Influence  Judgment,  and  about  the 
degree  to  which  people  (ordinary  folks  and 
experts)  are  calibrated  in  their  Judgments  about 
uncertain  events.  Reference  [8]  is  a  collection 
of  thirty  papers  with  over  650  reference*. 

Several  etate-of-the-art  papers  are  "must"  reading 
for  cost  uncertainty  methodology  developers  and 
user*.  Among  them  i*  "Calibration  of 


Probabilities:  The  State-of- the-Art  to  1980"  [11] 
and  "Encoding  Subjective  Probabilities:  A 
Psychological  and  Psychometric  Review"  [22].  A 
particularly  relevant  finding  ia  that  people  are 
not  well  calibrated  relative  to  rare  events:  in 
most  studies  there  is  strong  evidence  that 
decision  makers  systematically  underestimate  tail 
probabilities.  Thus  it  may  happen  that  actual 
costs  exceed  corresponding  elicited  95th 
percentiles  one  time  in  five  rather  than  one  time 
in  twenty.  It  is  not  too  surprising  that 
calibration  is  less  precise  in  the  tails  of  a 
distribution  than  in  the  fleshier  part*  since  we 
haven't  had  as  much  experience  with  rare  events  as 
with  nominal  events.  Alternately,  even  if  the 
decision  maker  is  viewed  as  an  ideal  data 
processor,  estimates  of  large  and  small 
percentiles  from  data  have  larger  variances  than 
estimates  of  aore  central  percentiles  in  general. 
If  this  empirical  observation  holds  true  with  cost 
analysts,  then  requiring  them  to  specify  h  and  1 
of  a  beta  distribution  l the  most  extreme 
percentiles)  would  tend  to  result  in  a  systematic 
understatement  of  uncertainty.  It  would  be  unfair 
to  criticize  the  developers  of  PERT  or  the  early 
cost  researchers  who  modified  PERT  methodology  to 
deal  with  cost  uncertainty.  Requiring  (l,m,h) 
assessments  was  a  matter  of  mathematical 
expedience  and  the  behavioral  research  did  not 
exist  then.  The  fact  that  the  112  references  in 
the  cost  uncertainty  state-of-the-art  paper  [15] 
and  the  550  references  in  [8]  have  no  common  entry 
can  be  viewed  as  an  exciting  opportunity  to 
improve  cost  uncertainty  methodology  by  exploiting 
a  relevant  body  of  related  research. 

There  are  no  theoretical  reasons  why  the  beta 
family  ought  to  enjoy  such  a  position  of 
prominence  in  the  cost  uncertainty  liturgy.  Nor 
is  there  any  particular  reason  for  avoiding  its 
use.  Because  of  its  flexibility  in  shape  and 
history  of  application,  it  will  probably  continue 
to  be  used  in  cost  uncertainty  analysis.  That 
being  the  case,  it  may  be  worthwhile  to  diepel 
some  common  misconception#: 
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a)  It  take*  4  conditions  to  specify  a  beta 
density . 

b)  The  "asymmetry  quotient"  AQ  ■  (h-m)/(m-l) 
is  good  measure  of  skewness. 

c)  Since  cost  is  bounded  above  and  below,  the 
beta  family  with  its  finite  range  is  better 
suited  to  represent  cost  uncertainty  than 
are  distributions  with  infinite  range. 

Assertion  a)  is  correct ,  of  course,  but  only  in  a 
strict  sense.  Suppose,  instead  of  (l,m,h)  plus  a 
4th  condition,  some  less  extreme  assessments  are 
made,  say  the  three  quartiles  are  elicited  (23th, 
50th,  and  75th  percentiles).  These  are  more 
natural  quantities  to  think  about  [6]:  for 

example,  they  break  the  range  of  costs  into  4 
equally  likely  regions.  Standard  elicitation 
techniques  require  relatively  simple  scaling  tasks 
with  even,  two-to-one,  or  three-to-one  odds. 

These  three  inputs  have  the  effect  of  tying  the 
cumulative  distribution  function  (CDP)  down  at 
three  points  called  "knots."  While  there  are 
infinitely  many  beta  distributions  whose  CDF's 
pass  through  the  knots,  the  functional  fora  of  the 
beta  family  has  the  effect  of  making  these 
different  distributions  practically 
indistinguishable.  Who,  but  God,  is  so  finely 
calibrated  in  his  cognitive  abilities  as  to  be 
able  to  distinguish  between  the  four  beta  CDF's 
(shown  dotted)  and  the  gamma  CDF  (shown  solid)  in 
figure  27  Corresponding  densities  are 
superimposed . 

It  is  hard  to  think  of  a  practical  situation 
in  which  the  to  .ai  cost  uncertainty  distribution 
would  be  effected  by  variation  in  component 
distributions  as  negligible  as  those  shown.  Thus, 
the  four  parameter  beta  distribution  is,  for  most 
practical  purposes,  a  three  parameter  family  -  as 
long  as  one  chooses  the  correct  three  parameters, 
here  percentiles,  to  assess.  Similar  results  hold 
for  other  choices  of  knots.  By  contrast,  fixing 
(l,m,h)  and  varying  s  over  unimodal  beta  distri¬ 
butions  produces  shapes  ranging  from  something 
nearly  flat  (uniform)  to  a  virtual  spike  at  m  in 
the  limit.  The  limiting  case  for  the  beta  with 
three  fixed  knots  is  a  well  behaved  gamma. 


t.O  «.J  1C.0  10.5  11.0  M.S  12.0 
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FIGURE  2:  Knotted  Beta  and  Gamma  CDF'S: 

Distributions  with  common  quartlles 

Figure  2  also  sheds  light  on  misconceptions 

b)  and  c).  The  four  nearly  indistinguishable  beta 
distributions  have  asymmetry  quotients  AQ  * 
(h-m)/(m-l)  ranging  from  A  to  69.  There  is  no 
relation  between  tnls  measure  of  '‘asymmetry”  [15] 
and  any  statistical  definition  or  cognitive 
concept  of  asymmetry.  It  appears  that  the 
asymmetry  quotient,  like  most  of  the  beta-fitting 
methodology,  was  Invented  for  mathematical 
convenience.  Its  cognitive  value  in  helping  cost 
analysts  quantify  uncertainty  is  nil.  Concerning 

c) ,  none  of  the  distributions  shown  in  figure  2 
have  any  meaningful  amount  of  probability  to  the 
right  of  12  (less  than  .00001).  Four  of  the  five 
distributions  (the  betas)  have  finite  upper  bounds 
ranging  from  about  twelve  to  over  sixty.  The 
fifth  (gamma)  has  an  infinite  right  tail.  For  all 
practical  purposes,  they  are  identical.  The  upper 
truncation  point  h  of  the  beta  density  is  Just  a 
parameter  which  obviously  has  very  little  to  do 
with  where  the  probability  mass  is  located 
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(another  argument  for  not  eliciting  h).  So  why 
should  the  cost  analyst  concern  himself  over 
inconsequential  issues  as  confusing  as  finite 
versus  infinite  range? 

Modifying  the  uncertainty  encoding 
techniques  of  PERT  to  fit  the  cost  analysis 
scenario  was  a  reasonable  first  step  in  1965  [19]. 
The  current  state-of-the-art  in  elemental 
uncertainty  quantification  os  described  in  [15] 
suggests  that  progress  in  the  last  19  years  has 
been  scant.  With  the  advent  of  inexpensive  and 
powerful  personal  computers,  computational 
complexity  is  no  longer  a  limitation  in  designing 
Interactive  software  for  probability  encoding. 
Cognitive  considerations  rather  than  mathematical 
expedience  should  dominate  the  search  for  new 
methodology.  The  challenges  are  exciting.  Some 
new  initiatives  are  discussed  in  [21]. 

2.  PROCESSING  OF  COMPONENT  UNCERTAINTIES 


Developing  an  uncertainty  distribution  over 
total  cost  or  over  some  intermediate  level  of 
disaggregation  such  as  WBS  cost  elements  is  a 
matter  of  applying  the  probability  calculus  to 
elemental  uncertainty  assessments  -  a  purely 
mathematical  task.  As  with  subjective  probability 
elicitation  methodology  and  software,  there  is  a 
large  gap  between  what  is  available  to  the  cost 
analyst  and  what  could  be  available.  A  parallel 
thrust  to  simultaneously  upgrade  both  elicitation 
and  processing  capabilities  is  needed:  state-of- 
the-art  processing  tools  won't  improve 
misspecified  uncertainty  distributions  (garbage 
in,  garbage  out)  nor  are  finely  honed  subjective 
assessments  worth  the  trouble  if  they  are 
improperly  processed.  Two  of  the  more  important 
processing  tasks  are  a)  uncertainty  associated  with 
parametric  costing,  and  b)  uncertainty  associated 
with  a  convolution  (the  summing  of  uncertain 
quantities  such  as  WBS  cost  elements). 

Parametric  cost  estimation  is  one  of  the  most 
widely  used  techniques  for  developing  point 
estimates  of  cost.  General  overviews  of  the  topic 
can  be  found  in  [10],  [13],  or  [17],  Empirical 
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model  building  techniques  (usually  regression 
analysis)  are  applied  to  historical  cost  data  on 
systems  Judged  more  or  less  analogous  to  the 
object  system  resulting  in  a  cost  estimating 
relation  (CER).  Point  estimates  and  prediction 
intervals  for  the  cost  of  the  object  system  can  be 
obtained  using  standard  regression  techniques. 

But  these  quantities  do  not  characterize  the 
uncertainty  distribution  for  the  object  system's 
cost.  For  concreteness,  consider  the  the  first 
unit  cost  CER  for  launch  operations  and  orbital 
support  (LOOS)  for  apogee  kick  motor  equipped 
satellites,  one  of  37  CER's  listed  in  [4]. 


fy79K$  ■  27.44  +  .2992x  (wet  wt  lbs) 

Stats:  n«12,  r-squared* . 80 ,  F-38.81 

std  error  (SE)  -  146.33  485 . 9Slbs<;3694 . 3 


The  CER  estimate  of  LOOS  cost  for  a  1000  pound 
unit  would  be  $326,640.  Assuming  the  model  is 
appropriate,  there  are  three  sources  of  uncer¬ 
tainty  to  recon  with:  1)  statistical  uncertainty: 
randomness  associated  with  the  estimates  of  the 
model  parameters,  2)  predictive  uncertainty: 
random  deviation  of  the  cost  of  the  object  system 
from  its  predicted  value,  and  3)  input 
uncertainty:  uncertainty  about  the  ultimate  value 
of  wet  weight  which  is  likely  to  differ  from  its 
expected  or  nominal  value  (1000  lbs).  This  last 
source  of  uncertainty  is  sometimes  called 
"strategic  uncertainty"  [3].  Failure  to  include 
any  of  the  three  may  result  in  substantial 
understatement  of  LOOS  cost  uncertainty.  Only 
input  uncertainty  is  treated  In  [13],  whereas  only 
input  and  predictive  uncertainties  are  included  in 
[24].  A  schematic  diagram  for  a  formal  cost 
uncertainty  analysis  is  shown  in  figure  3. 

All  of  these  sources  of  uncertainty  are  recognized 
in  [4]  but  not  dealt  with  explicitly.  Ad  hoc 


rules  replace  the  type  of  formal  analysis  shown  in 
figure  3.  The  output  is  in  the  form  of  (l,m,h) 


MODEL  ■  COST  -  a  +  b*(LBS>  ■*■£,€  -N(O.o') 

COST  analyst  tjputs  mathematical  analysis 


FIGURE  3:  CER  Uncertainty  Processing 

estimates,  not  an  uncertainty  distribution.  In 
addition,  since  1  and  h  are  computed  as  m  t  SE 
(i.e.  not  dependent  on  the  value  of  the  input 
variable),  their  raeani  \g  is  not  clear.  If  one 
wished  to  attempt  a  formal  uncertainty  analysis, 
the  form  in  which  the  data  has  been  summarized 
(see  above  stats)  is  incomplete  since  the 
sufficient  statistics  (sample  means  and 
covariances)  are  not  given. 

Direct  assessment  of  a  prior  joint 
probability  distribution  over  the  model  parametern 
is  cognitively  quite  difficult.  There  are  new 
indirect  methods  [7]  supported  by  interactive 
computer  codes  which  cay  prove  useful  if  adapted 
to  the  coat  analysis  scenario.  This  approach  ie 
discussed  in  (21]. 

The  final  step  in  generating  an  uncertainty 
distribution  over  total  cost  consists  of 
convolving  the  uncertainty  distributions  of  the 
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cost  elements  into  which  total  cost  was 
disaggregated  (probably  the  WBS ,  although  it  may 
be  advantageous  to  use  some  other  disaggregation 
scheme) .  It  is  tempting  to  take  the  bookkeeper 
approach  and  simply  sum  the  individual  estimates 
to  generate  a  "best"  estimate  of  total  cost.  This 
approach,  while  much  criticized  and  devoid  of 
mathematical  Justification  [12],  is  nevertheless 
widely  practiced.  The  CER  compendium  [4]  is  a 
case  in  point.  In  fact,  most  likely  subsystem 
costs  are  irrelevant  to  the  convolution  process. 
What  counts  is  their  uncertainty  distributions. 
Furthermore,  summing  most  likely  costs  will  result 
in  systematically  underestimating  total  cost  if, 
as  argued  by  some  [14],  only  right-skewed 
distributions  make  sense  in  quantifying  cost 
uncertainty.  While  there  are  undoubtedly  many 
reasons  why  cost  estimates  tend  to  be  low,  this 
one  is  easy  to  avoid  by  simply  using  correct 
methodology . 

The  two  moat  widely  used  methods  of 
convolving  component  uncertainty  distributions  are 
Monte  Carlo  simulation  (used  in  [2],  [6],  [0],  and 
[16]  )  and  a  cumulant  summation  technique  called 
the  method  of  moments(uaed  in  (14],  [19],  [23], 
[24],  and  [25]  ).  It  is  assummed  that  the  reader 

is  familiar  with  the  ideas  behind  these  two 
techniques.  The  output  of  the  Monte  Carlo  method 
is  sample  of  observations  from  the  total  cost 
uncertainty  distribution.  The  comforting  feature 
of  this  approach  is  that  the  empirical 
distribution  function  of  the  sample  data  converges 
to  the  "true"  total  cost  uncertainty  distribution 
as  the  number  of  replications  becomes  large.  By 
"true"  we  mean  the  distribution  that  is  the  exact 
convolution  of  the  component  uncertainty 
distributions.  With  today's  inexpensive  and 
powerful  microcomputere ,  the  expense  of  large 
samples  is  no  longer  a  limitation  of  this 
approach . 

The  output  of  the  cumulant  summation  method 
is  the  first  four  moments  of  ths  "true"  total  cost 
uncertainty  distribution  from  which  that 
distribution  mu*-  be  estimated,  Estimating  a 
distribution  from  four  moments  requires  an 


assumption  about  tha  family  to  which  the  "true" 
distribution  belongs  -  another  potential  source  of 
error.  But  unless  one  chooses  such  a  family, 
about  all  that  can  be  said  about  the  total  cost 
uncertainty  distribution  is  to  provide  probability 
bounds  [26]  based  on  these  moments.  These  bounds 
are  usually  fairly  broad  and  hence  of  limited 
practical  value.  The  degree  to  which  four,  (in 
this  case  [5],  even  five!)  moments  fail  to  nail  down  a 
distribution  is  shown  in  figure  4. 


^roee*  I  1 1 


-3  -2  -1  C  1  2  J 


FIGURE  4:  Tukey  Lambda  Densities.  Moments  up  to 
and  including  order  five  are  identical. 

While  cumulant  summation  methods  can  be  expected 
to  yield  reasonable  results  in  most  nominal  coet 
analyses,  it  is  difficult  to  tell  how  well  they 
will  work  in  any  specific  application.  In 
addition,  moment  methods  are  often  nonrobuat,  i.e. 
sensitive  to  small  changes  in  the  data.  Other 
comparisons  are  glv  en  in  [12].  For  these  reasons, 
as  methodologies,  the  Monte  Carlo  method  is  better 
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suited  to  cost  uncertainty  analysis. 

Ho  tatter  how  good  or  bad  a  methodology  is, 
poor  implementation  (algorithms,  computer  codes, 
ease  of  uee,  etc)  will  reduce  its  utility.  Two  of 
the  better  known  techniques  used  in  DOD,  one 
employing  a  Monte  Carlo  approach  [6]  and  one  using 
the  cumulant  summation  method  [14]  are  discussed  in 
[12]  and  [20].  Experience  with  actual  data  is 
reported.  Neither  approach  was  wholly 
uatief actory .  Por  example,  variance  calculations 
in  [6]  resulted  in  negative  numbers.  Using  the 
formulas  given  in  [14],  it  is  possible  to  fit  a 
Weibull  distribution  to  its  own  moments  and  come 
out  with  a  different  Weibull  distribution  as  shown 
in  figure  5. 


16  SO  54  It  33  36 


FIGURE  5:  Weibull  Densities.  The  output  density 
resultsd  from  applying  ths  formulas  of 
[11]  to  ths  moasnts  of  ths  input  dsnsity. 
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CONCLUSIONS 

A  cost  uncertainty  analysis  should  logically 
precede  the  choice  of  a  "best"  estimate  of  total 
cost.  Historical  evolution  of  the  cost  estimation 
process  has  resulted  in  a  reversal  of  these  two 
tasks.  Likewise,  the  PERT  approach  of  the  sixties 
seems  to  have  had  an  inhibiting  influence  on 
creativity'  in  developing  cost  uncertainty 
assessment  methodology.  Recent  advances  in 
psychometric  research  offer  hope  of  providing  the 
cost  analyst  with  applicable  tools  which  take  hie 
capabilities  into  consideration.  Development  of 
user-friendly  software  for  today’s  inexpensive  but 
powerful  microcomputers  will  facilitate  the  probability 
encoding  tasks.  There  are  exciting  opportunltee 
for  research  and  development  in  the  important  area 
of  cost  uncertainty  analysis. 
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